Repository navigation
Conversation
Completes the public-bind contract begun in #293. That change pinned the pre-flight refusal (a non-loopback bind without --allow-public-bind); this pins the other half, and the security property the coverage audit ranked first: after opting in, the endpoint must not be open. `rocm serve --host 0.0.0.0 --allow-public-bind` generates an API key, prints it exactly once with the `Authorization: Bearer` instructions, and hands the serving engine a key file to enforce. The scenario drives a real managed serve and then talks to the endpoint directly: a request with no key and a request with the wrong key must both be refused as unauthorized, and the key that was just issued must be accepted. GPU lane, because the ENGINE enforces the key (the CLI only issues and hands it over) and `serve` has no plan-only mode, so enforcement cannot be observed on the mock lane. The pure key-resolution rules — notably that a loopback bind stays credential-free even when --api-key is passed — are already unit-tested in the CLI, so this covers only what a unit test cannot: that a live public endpoint actually authenticates. Adds an endpoint_api_key field to the test World and exposes three existing serve helpers to the new step module. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The scenario served straight from a fresh isolated data directory, which has no ROCm runtime configured, so `rocm serve` refused before it could issue a key: "device_policy: gpu_required; no active ROCm runtime is configured". Every other GPU serve scenario opens with `Given a managed runtime is active`, which points the scenario at the runner's shared managed runtime; this one was missing it. Found by a scoped GPU dispatch of the scenario, which is also why the run needed reading rather than trusting: the self-hosted E2E lane is non-blocking, so the workflow run reports success even when the suite inside it fails. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
A managed serve returns once the supervisor is up, so for the first seconds the engine has not bound the port and the readiness probe's connection is refused. The probe treated any transport error as fatal, so the scenario died on that first refusal before it ever reached its assertions. Split the request helper: the readiness wait now retries until the endpoint answers (or the timeout expires), while the assertions keep treating a refused connection as fatal — by then the endpoint has already answered, so a transport failure there is a real fault rather than a startup race. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The scenario was tagged @requires-gpu only, so it also selected the native-Windows GPU lane. There the effective serve engine is lemonade, and the CLI deliberately refuses a public bind on Windows lemonade because the endpoint key cannot be enforced through it — a designed-in refusal, not a defect. The scenario would have driven that refusal and failed deterministically on that lane; the MI300X dispatch could not surface it because that host resolves to vLLM. Add @requires-engine:vllm, the same precedent serve-vllm-default-on-instinct uses. Because vLLM cannot start on Windows or on families that do not prefer it, the scenario now resolves to a skip on the Strix lanes rather than needing an expectations row for behaviour that is working as designed. It also keeps this cold serve off the time-capped Strix lanes. Also correct the show-once comment: the key does legitimately appear a second time inside the curl example line, so the assertion counts the dedicated `api key:` line rather than occurrences of the key. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
A run on MI300X timed out after the full readiness budget with nothing but "did not answer an authenticated request within 300s", which cannot distinguish the three failures that look identical from outside: the engine never started, it is still loading weights, or it is up and rejecting the key it just issued — the last of which would be a real defect in the contract under test rather than a slow start. Carry the engine's own log tail, the serve stdout, and the last observed endpoint response into the timeout panic, the same diagnostics the other serve steps already collect. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…nd-selectors Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/tests/e2e.rs
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · f75e4ee
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
Pins the "missing/invalid credentials are rejected" criterion of EAI-7409, which now resolves to ROCMAI-318. It does not cover the ticket's other criteria (secrets not logged or persisted insecurely, status/log protection), and it does not claim to.
Blocking
- (new) The new scenario's title breaks the required
feature_namingtest, so CI is red on Linux and Windows —tests/e2e-cucumber/features/networking.feature:64- The scenario is titled
Scenario: 4 - A public endpoint …. tests/e2e-cucumber/tests/feature_naming.rs:148requires per-feature sequential prefixes, and the three scenarios above it usenetworking-01tonetworking-03.- I ran
cargo test -p e2e-cucumber --test feature_namingin the checkout. It exited 101:scenario_names_are_indexed_sequentially_per_featurepanicked withnetworking.feature: scenario 4 is named "4 - A public endpoint issues a key once and then enforces it" but must start with "networking-04 - ". - The same panic is the only failure in both "Test (affected crates)" (job 120489475310) and "windows-build-and-test" (job 120489475455). The last three main runs of that workflow are green.
- Confidence 100 · mechanical · Fix: rename it to
Scenario: networking-04 - A public endpoint issues a key once and then enforces it.
- The scenario is titled
Non-blocking
- (new) The endpoint URL hardcodes the shared serve port instead of using the existing constant —
tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:25ENDPOINT_URL = "http://127.0.0.1:11435/v1"copies the value owned bySERVE_PORT(serving_steps.rs:172) andSERVE_BASE_URL(serving_steps.rs:78).- If the port moves,
ensure_serve_port_freefrees the new port while these steps keep probing 11435. - Separately, the step sets
world.endpointandworld.model_name(lines 119 and 121), but the Then-steps read the constant, notworld.endpoint. - Confidence 75 · mechanical · Fix: make
SERVE_BASE_URLpub(crate)or build the URL fromSERVE_PORT, and have the Then-steps readworld.endpoint.
- (new) The new serve step lacks the relaunch and failure evidence its sibling has —
endpoint_auth_steps.rs:99-122, compared withserving_steps.rsserve_and_wait/serve_gpu_model- It launches through
run_rocm_okwith a single attempt. - The sibling path reports a non-zero exit together with the engine log, and it has a budgeted relaunch. Per its own comment, that relaunch exists because this model on MI300X has both timed out and served fine within the same run.
- Here, a non-zero exit panics without the service-log tail. Only the timeout path attaches it.
- Reusing
serve_and_waitas-is would not work, because its readiness probe is unauthenticated and would get 401. So this is a matter of extending the shared helper, not swapping it in. - Confidence 70 · logic.
- It launches through
- (new) The feature's header comment is now inaccurate —
tests/e2e-cucumber/features/networking.feature:3-9- It says these scenarios "run on the mock (no GPU) on every platform the suite runs on".
- The new scenario is
@requires-gpu @requires-engine:vllmand runs only on vLLM-capable dcgpu hosts. - The scenario-level comment is correct; the header is what is now stale.
- Confidence 75 · mechanical · Fix: qualify the header, e.g. "all but networking-04".
Decisions for the author
- A cold vLLM serve on the per-PR MI300X lane, versus pinning enforcement at all — tradeoff
- Enforcement lives in the engine (
ROCM_SERVE_API_KEY_FILE→VLLM_API_KEYatengines/vllm/src/process.rs:256), and nothing else tests that path: there is no unit test touchingVLLM_API_KEY. So a GPU scenario is the only place this can be proven. - The cost is roughly 65 s of serial MI300X time per run (observed: 16:39:36 to 16:40:41). The scenario comment argues the choice, and this note is only to confirm it was deliberate.
- Enforcement lives in the engine (
Positive signals
- The readiness wait polls with the issued key and treats a refused connection as "not yet". That means an open endpoint (200 for any bearer) cannot satisfy the 401 assertions, and a stray loopback engine on 11435 produces a red result, never a false green.
- The timeout panic carries serve stdout, the service-log tail and the last endpoint status. That separates "never started" from "still loading" from "rejecting its own key".
Deployment notes
None
What this covered
- What was read: every changed file in
prw-base...f75e4ee6, read completely. Also: the CLI code that prints and hands over the key (service_launch.rs,serve_cmd.rs), the vLLM adapter's key handling,E2eWorld's Drop teardown, the expectation and tag resolution for@requires-engine, andruntime_steps.rsfor the Given step. - Verdicts on the added tests:
- Would it pass with the behaviour reverted? No. The PR changes no production code; it pins existing behaviour. By reading, if the key stopped reaching vLLM, the readiness wait would pass and the unauthenticated step would get 200 instead of 401, so the scenario fails.
- What stays green if one branch breaks? Nothing found. A wrong-key-accepted engine fails the wrong-key step. A key printed but not passed to the engine times out in readiness. A second dedicated
api key:line fails the show-once step.
- I did not run either mutation: the scenario needs a GPU and none was available here. vLLM's guarding of
/v1/*is from knowledge of vLLM, not from source in the checkout.
- Lane gating, cleanup and CI:
- Gating checked: the scenario skips on Strix Windows, Strix Ubuntu and W7900, and runs on MI300X.
- Cleanup checked:
E2eWorldDrop runsrocm services stopeven when a step panics, so the 0.0.0.0 bind lasts no longer than any other serve. Noexpectations.tomlrows are affected. - On "E2E tests (MI300X, current)" the new scenario ran and passed (6/6 steps). The job reports 246 scenarios (240 passed, 6 failed). The failures are at
chat.feature:70,model_serving.feature:11/16/68,runtime_setup.feature:32andtherock_next_generation.feature:166. Main's MI300X run on the PR's base 1c33248 (run 39975715090) fails at exactly the same six locations with the same count. Five of the six are declared xfail; the one unexpected failure,therock-next-09(a vLLM/amd-aiter dependency conflict), is the same on main. None of it is attributable to this PR.
- Local runs:
cargo checkandcargo clippy -D warningson the e2e target are clean (exit 0). - Passes that did not run:
- Prior-changes pass: earlier reviews are out of bounds for this run.
- History: only a light check (recent
git logon the feature file).
- Fan-out: two independent workers, one for the code pass and one for CI attribution. I verified their findings myself.
- Discussion: not reconciled against the PR's comments or reviews.
…ve URL Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Requested changes addressed: commits were pushed after this review and all review threads are resolved. Re-review requested.
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 02726d1
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Round 2. No change request filed. The earlier change request's blocker was the networking-04 title failing feature_naming. It was standing, not re-found blind, and was settled by putting it to the reviewer after this review: the title now reads networking-04 - …, feature_naming passes 4/4 locally, and "Test (affected crates)" and "windows-build-and-test" are green at this head. The one new blocking-class finding below (a stale sentence in docs/testing.md) understates coverage rather than overstating it, so the automation reports it without gating on it. The only red check, "E2E tests (MI300X, current)", has the same 5 xfail + 1 unexpected failure (therock-next-09) as main's run at the base 1c33248; networking-04 passed all 6 steps there.
Review — needs work
Full review of the whole change.
This PR adds end-to-end evidence for one acceptance criterion of EAI-7409 (now ROCMAI-318): "Missing/invalid credentials are rejected". It does not claim to close the ticket, which is still In Progress.
Blocking
- (new)
docs/testing.mdstill says this assertion is not exercised anywhere —docs/testing.md:955-958
The doc says: "Verifying that the running server actually rejects an unauthenticated request requires a live engine; that end-to-end assertion is a deferred follow-up and is not yet exercised by the GPU acceptance scripts."networking-04is exactly that assertion, so the doc became false when this PR landed the scenario. AGENTS.md §5 requires docs/testing.md to be updated in the same change and the claim to be checked wherever it repeats. The code is the side that governs here: the scenario exists and passed on MI300X at this head.
Confidence 85 · mechanical · Fix: rewrite the sentence to namenetworking-04(features/networking.feature) and say it runs on the MI300X vLLM lane only, which is non-required. Native Windows and lemonade-default hosts skip it.
Non-blocking
-
(standing, expanded)
serve_public_bindbrings back the serve-then-poll pattern thatserve_and_waitwas written to replace —tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:84-102,:108-129- What the in-file guidance says: the doc comment on
serve_and_waitinserving_steps.rssays it "Replaces therun_rocm_ok+ readiness-poll pair", and adds "A non-zero exit is reported the same way rather than throughrun_rocm_ok, since the engine log explains those launches too." - Lost failure evidence: the new step calls
crate::run_rocm_okand discards the device-state string thatensure_serve_port_free()returns. Its timeout panic carries no device state, stderr, archived log or stop status (serve_failure_evidence). - Lost readiness guard: it returns on any 2xx and drops
host_serve_target().2as_.model_is_ready's doc warns that a plain 200 on the shared port 11435 can come from a leaked serve. In that case the "without the key is refused" step would fail with a message that looks like a security defect, when the real cause is a harness leak. This is a misleading red, not a false green. - Commit message: 039d0e4 claims "the same diagnostics the other serve steps already collect", which is not accurate. The repo squash-merges, so the squash message should not repeat that claim.
Confidence 85 · architectural · Fix: add an optional bearer token to
model_is_ready/serve_and_wait(and have it return stdout), or makeserve_failure_evidencepub(crate). Then route both the non-zero-exit and timeout paths through it, and require the ready-substring in the authenticated/modelsbody. - What the in-file guidance says: the doc comment on
-
(standing) The feature header comment is now partly stale and not re-wrapped —
tests/e2e-cucumber/features/networking.feature:6-9
Line 6 still says the scenarios drive the binary "against the mock server", and the feature's stated purpose is the CLI's native HTTP stack. networking-04 runs on a real engine, and its HTTP checks go through the harness'sreqwest, not the CLI. The edited line 8 is 105 characters while its neighbours wrap at about 80.
Confidence 90 · mechanical · Fix: re-wrap the header, and say networking-04 is a GPU serve scenario that checks enforcement from the harness.
Decisions for the author
- (standing) The scenario only runs on a non-required lane — tradeoff
@requires-engine:vllmlimits it to MI300X..github/workflows/e2e-selfhosted.ymlsays that check "isn't in the required list" and skipsmerge_group, so a regression in key enforcement would not block a merge. The lemonade-on-Linux enforcement path (--api-key-file) has no scenario. The tag is well argued (Windows lemonade refuses a public bind by design) and the PR text names the lane. Confirm you accept that this contract is guarded only in an advisory way. - (new) What "issues a key once" proves — non-blocking-improvement
shows_key_oncecounts lines that start withapi key:. It would stay green if the key leaked onto any other line. It also never checks the documented promise that the key does not appear inrocm servicesorrocm logs. Possible fixes:- assert
output.matches(key).count() == 2(theapi key:line plus the curl example), or - add a step that lists services and checks the key is absent.
- assert
Positive signals
try_models_statusandget_models_statuskeep two cases apart: "no listener yet" is retried during readiness, and "answered with a status" is asserted. The timeout panic names the three cases that look identical from outside, including the one where the engine rejects the key it was issued, which would be a real contract defect.- The
@requires-engine:vllmcomment explains why this is a tag rather than an xfail row, and the cited facts check out:ensure_public_bind_engine_supportedandresolve_endpoint_auth_loopback_stays_credential_freeexist as described.
Deployment notes
None
What this covered
- Read: all four changed files in full, against
prw-base1c33248…02726d1a. Also the surrounding harness (serving_steps.rsserve/readiness helpers,run_rocm_ok, capability tags, the World cleanup), the CLI's public-bind and key-printing code, and how the vLLM and lemonade engines receive the key. The work was split across two independent read-only workers (code pass; agent-instruction/history/comments/commits pass), plus my own verification. - Run locally:
cargo test -p e2e-cucumber --test feature_naming: 4/4 passed.cargo clippy -p e2e-cucumber --all-targets -- -D warnings: clean.- Sign-off check: all 6 commits signed off.
- CI at the head:
- Every check passes except "E2E tests (MI300X, current)".
- In that job,
networking-04passed all 6 steps. - The job's 6 failures are the same 6 scenarios that fail on main's run at this base SHA. They are pre-existing, not caused by this PR.
- Step verdicts:
- The "without the key" and "wrong key" steps both go red if the engine stops enforcing the key.
- The "issued key is accepted" step adds nothing beyond the readiness wait.
- Could not run: GPU scenarios (no GPU here), so the GPU evidence above is CI's, not mine. Prior-changes pass: no earlier review comments were read, by design.
- Not reconciled: this report was not checked against the PR's comment discussion.
…k, fix feature header Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Fixed in 9008999. docs/testing.md now names networking-04 and says it runs on the MI300X vLLM lane only (non-required; Windows/lemonade-default hosts skip it). The key-once step now fails if the key appears on any line other than the api key line and the curl example. The networking.feature header is re-wrapped and says networking-04 is a GPU serve scenario checked from the harness. Not done, left for a follow-up: routing serve_public_bind through serve_and_wait/serve_failure_evidence needs a shared-helper change (bearer token in model_is_ready), so it stays out of this PR. The non-required-lane tradeoff is accepted as documented. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 9008999
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Round 3. No change request filed. Last round's blocking-rated item (the docs/testing.md sentence calling the end-to-end auth check a deferred follow-up) was standing, not re-found blind; it was settled by putting it to the reviewer after this review: the sentence is gone, the new text names networking-04 and matches the scenario's tags, and no other surface repeats the old claim. The one blocking-class finding below (a bare backtick path where AGENTS.md §5 asks for a relative link, to a file that exists) is reported without gating, consistent with earlier rounds in this repo. The only red check, "E2E tests (MI300X, current)", has the same result as main's run at base 1c33248 (5 xfail + 1 unexpected failure, therock-next-09); networking-04 passed all 6 steps there, so it is not attributed to this PR.
Review — needs work
Full review of the whole change.
Adds end-to-end coverage for the "missing/invalid credentials are rejected" acceptance criterion of EAI-7409. It does not claim to close the ticket, and the ticket stays open (In Progress), which is consistent with that.
Blocking
- (new) The new
docs/testing.mdcitation uses a bare path instead of a relative link —docs/testing.md:956
The added text cites the feature file as(`tests/e2e-cucumber/features/networking.feature`). AGENTS.md §5 says a citation you add or edit should be a link relative to the citing file, not a bare backtick path. The reason given is that thedocs-linksCI job and the lychee hook can only catch a moved or deleted target when it is a link. The same document already does this for sibling feature files, for example[runtime_setup.feature](../tests/e2e-cucumber/features/runtime_setup.feature)at line 527 and[comfyui.feature](...)at line 700.
Confidence 85 · mechanical · Fix:([networking.feature](../tests/e2e-cucumber/features/networking.feature)).
Non-blocking
- (standing, re-found blind in a new form) The new serve step skips the stall relaunch that the other vLLM serve on this lane relies on —
tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs(serve_public_bind); for comparison,tests/e2e-cucumber/tests/e2e/serving_steps.rs(serve_gpu_model)
serve_public_bindserves the same target,Qwen/Qwen3.5-0.8Bon vLLM viahost_serve_target(). It makes one launch and one wait.serve_gpu_modeldeliberately allows one relaunch for that same model, with the comment "Qwen3.5 has both timed out and served successfully in the same MI300X run". Its retry also callsserve_failure_evidence/rocm services stopfirst, so a stalled attempt does not keep holding device memory. The new scenario therefore keeps a flake source that the established serve path was written to absorb. It also panics throughrun_rocm_okon a non-zero launch, so the service-log evidence thatserve_and_waitattaches is missing. The lane is not required, which limits the impact.
Confidence 70 · logic · Fix: route the launch through the existing serve-and-wait/relaunch machinery, passing the extra--host 0.0.0.0 --allow-public-bindargs and a key-bearing readiness probe, instead of a parallel copy. - (new) A key the engine rejects only shows up after the full serve timeout —
tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs(wait_for_authenticated_endpoint)
A 401 to the issued key is an answer from a live, bound engine. The doc comment itself calls it "a real defect in the contract under test, not a slow start". Even so, the loop keeps polling until the deadline, which spends serial GPU-lane time before reporting a definite failure. Failing fast on401would report that defect immediately. The loop also does not check the model id, asmodel_is_readydoes, but the later 401 assertions would catch a wrong server loudly, so this is efficiency, not correctness.
Confidence 70 · logic · Fix: return a failure as soon as the issued key gets401 Unauthorized.
Decisions for the author
None
Positive signals
- The readiness probe uses the issued key and tells a transport failure (
None) apart from an HTTP status. Only the assertions treat a transport failure as fatal, so an engine that is still binding is not mistaken for a refusal. - The scenario has three independent signals: no key gives 401, a wrong key gives 401, and the issued key is accepted. Removing key enforcement from the engine hand-off would fail it, not pass it.
@requires-engine:vllmis the right gate. The harness skips on hosts where vLLM cannot start, andeffective_serve_engineuses the same family predicate. So wherever the scenario runs,host_serve_target()resolves to vLLM, and the documented Windows/lemonade refusal is never driven.
Deployment notes
None. The change is test- and docs-only.
What this covered
Read completely: every file in the change, plus the code around it. That includes the serving_steps.rs serve helpers (serve_gpu_model, serve_and_wait, model_is_ready, ensure_serve_port_free), E2eWorld's Drop (which stops managed services on panic, so a timed-out serve does not leak), the @requires-engine skip logic in expectation.rs and capability.rs, and the CLI's key-disclosure printers in apps/rocm/src/service_launch.rs and serve_summary.rs. Diff prw-base (1c33248)…90089998.
Checks run:
- CI rollup on the head: one check failed, "E2E tests (MI300X, current)". Its log shows
networking-04ran and passed all 6 steps. The only unexpected failure istherock-next-09, whererocm engines install vllm --reinstallhit an unsatisfiable vLLM/amd-aiter pin. This change does not touch that path, so I did not count it against the PR. - "Sphinx docs build (-W)" and "Skill checks (skillscope)" were skipped. The diff touches neither Sphinx sources nor skills.
- The sign-off check returned exit 0: 7 commits checked, none missing.
What did not run, or ran differently:
- I did not fan out. The change is small enough for one context to read completely, so I used a single pass. The agent-instruction, history and code-comment passes ran in that same context, so they were not independent.
- Prior-changes pass: not run, because earlier reviews are out of scope for this run.
- Nothing was built or executed locally. The scenario needs a vLLM-capable GPU, and CI's MI300X run on this head stands in as the run evidence.
The PR discussion was not reconciled here.
…issued key gets 401 Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Fixed in 9567603. The docs/testing.md citation is now a relative link to networking.feature, like the sibling feature links. wait_for_authenticated_endpoint now fails immediately when the issued key gets 401, with the serve stdout and service-log tail, instead of polling to the deadline. Not done: routing serve_public_bind through the serve_and_wait relaunch machinery needs a shared-helper change (key-bearing readiness probe plus extra serve args), so it stays out of this PR as a follow-up; the lane is non-required. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 9567603
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — no blocking findings
Full review of the whole change.
Adds end-to-end proof for one acceptance criterion of EAI-7409 (now tracked as ROCMAI-318): a public bind rejects a missing or wrong credential. It does not cover the "secrets are not logged or persisted insecurely" criterion, and it does not claim to close the ticket.
Since the last round (9008999): both items it raised against the previous head are fixed here:
docs/testing.mdnow linksnetworking.featureinstead of citing it as a bare path.- The readiness wait fails immediately when the issued key gets a 401.
Each finding below is marked new or standing.
Blocking
None
Non-blocking
-
[standing] The new serve step brings back the launch-then-poll pattern that the shared serve helper says it replaced —
tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:84-136, againsttests/e2e-cucumber/tests/e2e/serving_steps.rs:133-160and:555-590
The doc comment onserve_and_waitsays it "Replaces therun_rocm_ok+ readiness-poll pair", because that pair loses the evidence needed to explain a serve that stalls.serve_public_bindis exactly that pair. Compared with the shared path it is missing four things:- Non-zero exits: it goes through
run_rocm_ok, so a non-zero exit panics without the engine log. - Failure evidence: it does not call
serve_failure_evidence,archive_service_logor stop-before-report, and it throws away the device state thatensure_serve_port_free()returns. - Relaunch: it has no one-relaunch rescue. The shared helper's comment says the same model on the same engine ("has both timed out and served successfully in the same MI300X run") needs that rescue.
- Model check: its readiness wait accepts any 2xx and never checks the model id.
model_is_readychecks it to guard against a leftover server from an earlier scenario. Without the check, a leftover unauthenticated vLLM would make the next step fail with a misleading "must be refused, got 200". That gives a false red, never a false green.
Confidence 85 · architectural · Fix: add an optional bearer token to
model_is_ready/serve_and_waitand let the caller get stdout back to extract the key, rather than keeping a parallel launcher. - Non-zero exits: it goes through
-
[new] The docs say the scenario runs on "the MI300X vLLM lane only", but more lanes run it —
docs/testing.md:958
.github/workflows/nightly.ymlalso runs the full suite, including this scenario, on "E2E tests (MI300X, incl. nightly-only)" (line 365) and "E2E tests (MI350P, incl. nightly-only)" (line 546). gfx950 passesengine_available("vllm"), so@requires-engine:vllmdoes not skip it there. The workflow file is what decides where the scenario runs, so the docs sentence is the side to correct. The PR description repeats the claim ("runs only on MI300X").Confidence 85 · mechanical · Fix: name the vLLM/Instinct lanes, for example "the per-PR MI300X lane, plus the MI300X and MI350P nightly lanes".
-
[new] The step checks the "shown only now" promise as wording only —
tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:175-178
AGENTS.md §3 says a promise that something will not happen is asserted together with the resulting state. Where the text is pinned on its own, the assertion should carry a comment naming the test that proves the behavior. This step checks the phrase and that the key does not leak elsewhere in launch stdout. It does not show the key is absent fromrocm servicesorrocm logs, which the docs paragraph it edits also promises, and it carries no such comment. The engine is already up at that point, so arocm services/rocm logscheck would be cheap.Confidence 70 · logic · Fix: add those two checks to this scenario, or add a comment pointing at the test that proves the promise.
Decisions for the author
-
Pinning the scenario to vLLM leaves the Linux lemonade and llama-server key paths with no end-to-end coverage — tradeoff
- What the pin buys:
@requires-engine:vllmstops the scenario from driving the Windows-lemonade refusal, which is intended behavior, and keeps a cold serve off the time-capped Strix lanes. - What it costs: on Linux the key also reaches lemonade (
LEMONADE_API_KEY,engines/lemonade/src/process.rs) and llama-server (--api-key-file). Those are separate enforcement paths that run on the Strix Ubuntu, WSL2 and W7900 lanes, and no scenario checks them. - The alternative: gating on OS instead of engine would cover them, at the cost of slower lanes.
Worth stating in the PR that this choice was deliberate.
- What the pin buys:
-
The public bind on a shared runner, and the issued key printed in panic text — tradeoff
The scenario binds 0.0.0.0 on a self-hosted runner, so while it runs the endpoint is protected only by the behavior under test. Several failure messages print the full serve stdout, key included, into CI logs and report artifacts of a public repository. The key is random per run, and teardown (the world'sDrop) stops the service, so the exposure is short. Redacting the key in panic text would be cheap.
Positive signals
- The readiness wait fails immediately on a 401 to the issued key, rather than waiting out the deadline. That separates "the endpoint rejects the key it issued" from "the engine is still loading".
- The leak check counts only the dedicated
api key:line and allows only theexample:curl line, so it would catch the key appearing anywhere else in launch output. That matches the real output fromrender_endpoint_client_configinapps/rocm/src/service_launch.rs.
Deployment notes
None
What this covered
Every changed file read completely: docs/testing.md, networking.feature, e2e.rs, endpoint_auth_steps.rs and serving_steps.rs. Diffed prw-base...HEAD, merge-base d510680…95676038. Also traced:
- the CLI's key printing (
apps/rocm/src/service_launch.rs,serve_cmd.rs); - the vLLM and lemonade key hand-off (
engines/*/src/process.rs); - the
@requires-engine:tag handling (tests/e2e-cucumber/src/expectation.rs,capability.rs); - teardown, and the nightly and self-hosted workflow lane definitions.
Two workers did the code and design passes. Runs in the checkout's own target dir:
cargo test --test feature_naming: 4 of 4 passed.cargo clippy --test e2e -- -D warnings: clean.
CI at 9567603:
- Hosted and required checks: green.
- Strix Halo, Windows lane: failed in pre-warm, before any scenario ran. A third-party download returned HTTP 403 while installing the lemonade backend, so this is not attributable to this PR. The same lane passed on main tip 63ecab2.
- MI300X and W7900 lanes: queued. MI300X is the only per-PR lane that runs the new scenario, so there is no CI evidence for it at this head. On the previous head 9008999, MI300X ran
networking-04and all six steps passed. That lane failed only on an unrelated vLLM reinstall scenario (therock-next-09). The difference from that head to this one is a docs link plus the 401 fail-fast branch.
Sign-off check: all 8 commits carry a sign-off trailer (exit 0).
Did not run:
- The live GPU scenario itself.
- vLLM's status codes on the pinned version. That vLLM answers 401 on
/v1/*is the workers' knowledge of upstream vLLM, not something traced in this repo. - Live branch-protection required checks, so "not a required check" in the docs is unverified.
- The history and prior-changes passes ran with git history only; no earlier review comments were read.
Not reconciled against any discussion here.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 9567603
Scope: this PR is test-only, adding/expanding tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs (~223 new lines) to pin the public-bind endpoint authentication contract, plus small visibility/doc changes elsewhere. I reviewed the diff against main and cross-checked the step assertions against the actual CLI source (apps/rocm/src/service_launch.rs::render_endpoint_client_config, apps/rocm/src/serve_cmd.rs summary-mode branch) to confirm the matched strings ("api key:", "shown only now", "Authorization: Bearer", the "example:" line) are accurate for a piped/non-TTY invocation. I also checked that vLLM's API key is loaded before the listener starts, to rule out a possible 401-race false positive in the readiness wait.
What I could not verify: there is no Rust toolchain in this sandbox, so I did not build the crate or run the e2e/cucumber suite — this is a static reading of the diff only, not an executed verification.
Findings (non-blocking; test-quality/coverage items, no correctness bugs found):
-
(low-medium confidence, non-blocking)
endpoint_auth_steps.rs~line 137: the no-key-leak assertion (shows_key_once/ similar) only scans captured stdout. Ifserve_public_bindonly stores stdout intoworld.cli_outputand discards stderr, a future regression that logs the API key to stderr (e.g. viaeprintln!or a tracing/log warning) would not be caught, even though the test's stated purpose is pinning that the key must "never appear ... in the audit log" more broadly. Consider also capturing/asserting over stderr if the harness can combine both streams. -
(low confidence, non-blocking/cleanup) ~line 47: a new
reqwest::Clientappears to be constructed on every call to the status-check helper (try_models_status), rather than being built once and reused (e.g. viaLazyLock, mirroring the existingSERVE_BASE_URL/MODELS_URLpattern). Since this is invoked in a polling loop (up to ~600s at 5s intervals) plus from eachThenstep, reusing a client would avoid repeated connection-pool/TLS setup overhead. Minor, since this is test code on a presumably already-slow serial e2e lane. -
(low confidence, non-blocking/cleanup) ~line 164: the models URL appears to be rebuilt as
format!("{}/models", endpoint_url())from the newly-exportedSERVE_BASE_URL, duplicating the existingMODELS_URLstatic already defined inserving_steps.rs(format!("http://127.0.0.1:{SERVE_PORT}/v1/models")). Two independent constructions of the same URL risk silently diverging if the path changes in only one place. Consider reusing the existing static instead.
Overall: no blocking findings — the new assertions appear to correctly pin the intended auth-output contract based on static reading of the diff and cross-reference against the CLI source producing that output.
🤖 Generated with Claude Code
|
🤖 Automated review (Claude Code) · 9567603 I reviewed the new e2e step definitions added for the public-bind endpoint authentication scenario, focusing on Two non-blocking diagnostics gaps stood out, both consistency issues rather than correctness bugs: (1) I did not have a GPU/vLLM toolchain available to actually run the new scenario, so I could not verify the live 401/401/2xx behavior end-to-end; this review is based on static reading of the diff and surrounding test infrastructure only. 🤖 Generated with Claude Code |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 9567603
Review — no blocking findings
Full review of the whole change (merge-base d5106802…95676038), with two independent fan-out passes (code-correctness, design/approach) plus a direct cross-check of the CLI/engine source the new test asserts against.
Implements acceptance criterion 2 ("missing/invalid credentials are rejected") of ROCMAI-318 (née EAI-7409) end to end; criteria 1 (anonymous 0.0.0.0 bind refused) and 3 (secrets not logged/persisted insecurely) are out of scope here and the change does not claim to close the ticket.
Blocking
None
Non-blocking
- CI rollup shows two self-hosted E2E lanes red, which the PR description does not reflect — PR description test-plan, vs. live rollup
The description's test-plan says "MI300X re-run on the current head is queued behind the serial runner" and only discusses a historical MI300X infra failure from 2026-08-25. The live rollup for the current head (gh pr view 722 --json statusCheckRollup) showsE2E tests (MI300X, current)andE2E tests (Strix Halo, Windows, current)bothFAILURE/COMPLETED(completed 2026-10-26), plus the derivedE2E consolidated report (self-hosted)alsoFAILURE— i.e. already-resolved results the description doesn't mention, not a pending queue. I could not fetch job logs (Actions API returned 403 for this token), so I can't confirm root cause. Note for calibration:.github/workflows/e2e-selfhosted.yml's own top-of-file comment states none of this workfow's check names are in branch protection's required-status-check list, and none usecontinue-on-error, so these are "real hardware regression shows red" signals by design, not merge gates — which is why this is non-blocking rather than blocking.
Confidence 100 · architectural · Fix: before merge, re-check live CI state (per this repo's ownAGENTS.md§4, "Live State Verification Before Any External Claim") and either update the test-plan checklist to match, or get/paste the actual MI300X and Windows job logs and confirm both failures are the known infra flake rather than something this change triggers. shows_key_onceonly scans captured stdout for key leakage, not stderr —tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:137-160
world.cli_outputis populated byrun_rocm_ok, which (pertests/e2e-cucumber/tests/e2e.rs:838-848) discards the child's stderr entirely. The test's own stated purpose is pinning that the key is shown exactly once and never leaks elsewhere, mirroringdocs/testing.md's broader "must never appear inrocm services,rocm logs, or the audit log" contract — but a future regression that prints or logs the key viaeprintln!/a tracing warning to stderr would pass this check undetected.
Confidence 85 · logic · Fix: capture/merge stderr into the same assertion (or add a parallel "stderr contains no secret" check) if the harness can combine both streams for this invocation.endpoint_url()'s/modelsURL is built independently of the existingMODELS_URLstatic —tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:29-31,47-50vs.tests/e2e-cucumber/tests/e2e/serving_steps.rs:80
Both ultimately derive from the sameSERVE_PORT, so practical divergence risk is low, but the/modelssuffix is now spelled out in two independent places (format!("{}/models", endpoint_url())here vs. the pre-existingMODELS_URLstatic) rather than reusing the latter.
Confidence 70 · mechanical · Fix: reuseMODELS_URL(nowpub(crate)-visible, likeSERVE_BASE_URL) instead of re-deriving the path.
Decisions for the author
- Driving the auth contract with a real HTTP client against a live GPU-backed vLLM engine, rather than a mock/stub standing in for "the engine" — tradeoff
This is the only way to prove the engine (not the CLI) actually enforces the key, which is the documented point of this scenario, and the pure key-resolution rules are already unit-tested. The cost is that the only end-to-end proof of acceptance criterion 2 runs on a single non-required GPU lane. Both the design and the gating follow established precedent elsewhere in this suite (e.g.serve-vllm-default-on-instinct,serve-vllm-inference), so this isn't a new gap this change introduces — flagging only so the tradeoff is explicit rather than assumed.
Positive signals
wait_for_authenticated_endpoint's failure diagnostics (serve stdout + service log tail on both timeout and an unexpected 401) directly address the class of bug this suite's own doc comments say previously made a similar scenario fail "before it ever reached its assertions" — good use of the existingservice_log_tailhelper rather than a bare panic message.- The
@requires-engine:vllmtag's rationale is spelled out in the feature file with the exact mechanism it's guarding against (Windows/lemonade can't enforce the key), rather than being a bare tag — makes the gating auditable instead of folklore. - Traced the key-issuance and enforcement path end to end (
service_launch.rs::render_endpoint_client_config→engines/vllm/src/process.rssettingVLLM_API_KEYon the command beforespawn()) and confirmed there is no startup race where the engine could answer before auth is configured, which is exactly the kind of flake risk a scenario like this could otherwise have.
Deployment notes
None — test-only change, no new runtime dependency (reqwest was already a tests/e2e-cucumber dependency), no schema/migration impact.
What this covered
Read the full diff (docs/testing.md, tests/e2e-cucumber/features/networking.feature, tests/e2e-cucumber/tests/e2e.rs, the new tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs in full, tests/e2e-cucumber/tests/e2e/serving_steps.rs) plus the surrounding/cross-referenced source: tests/e2e-cucumber/tests/feature_naming.rs, tests/e2e-cucumber/src/serve_log.rs, apps/rocm/src/service_launch.rs and serve_cmd.rs (output source for the asserted strings and summary-vs-plain dispatch), engines/vllm/src/process.rs/state.rs (key wiring), and .github/workflows/e2e-selfhosted.yml (required-check status). Ran two independent fan-out passes in fresh contexts (blind code-correctness; design/approach via targeted questions) in addition to this direct reading; both converged with my own findings and found no additional defects at or above the confidence bar. Cross-checked EAI-7409/ROCMAI-318's acceptance criteria against the change via the issue tracker.
Did not run: no Rust toolchain available in this environment, so nothing was executed directly — correctness here rests on static reading plus the repo's own passing Test (affected crates) / windows-build-and-test / clippy CI checks (taken at face value) rather than a local run. Could not fetch Actions job logs (403) to independently diagnose the two failing self-hosted E2E lanes beyond their rollup conclusion and the workflow file's own required-check documentation. Prior-review-comments pass not run (this review was reached blind, without reading this PR's existing review threads, per the methodology's independence requirement). Not reconciled against any discussion on this PR.
r0x0r
left a comment
There was a problem hiding this comment.
I went through this one and it looks good to me — approving, with two small inline nits that I'd be happy to see folded in or waved off.
It's a test-only change, and the scenario it adds (networking-04) passed 6/6 in about 61 seconds on the MI300X lane, so the new coverage genuinely runs. The lane shows red, but the failure is therock-next-09: a uv resolver conflict between amd-aiter==0.1.22 and 0.1.24 in the ROCm10 wheel index, which has nothing to do with this diff. That lane is green on main as of 2026-11-07 and isn't a required context, so I don't read it as a signal against this PR.
One thing I'd ask for rather than require: the head is now 12 days stale. A rebase plus a re-dispatched MI300X run would make the evidence current, and it's cheap insurance given how much of the value here is "the scenario actually passes on real hardware." Related — the PR body's test plan has drifted: it says the MI300X re-run "is queued" and cites failures from 2026-08-25, but a run on this head completed on 2026-10-28. Worth refreshing so the next reader isn't chasing stale results.
On the substance: the mid-readiness 401 fast-fail is the right call — it cleanly separates a genuine contract defect from a model that's merely slow to come up, instead of burning the whole timeout and reporting an ambiguous failure. Using the @requires-engine:vllm tag rather than an expectations.toml row also matches that file's own documented guidance, and the Gherkin itself reads as clean behavioural language rather than a transcript of the mechanics.
| Some(status) if status.is_success() => return, | ||
| // A bound engine rejecting the key it was issued is a contract defect, | ||
| // not a slow start, so report it now instead of at the deadline. | ||
| Some(reqwest::StatusCode::UNAUTHORIZED) => panic!( |
There was a problem hiding this comment.
Small one, non-blocking: both failure paths here skip the repo's own serve-failure evidence bundle.
This applies to the 401 fast-fail on line 91 and to the deadline assert! on line 100. Both panics do quote serve stdout plus service_log_tail, which is better than nothing — but unlike every other serve step, which routes through serve_failure_evidence (tests/e2e-cucumber/tests/e2e/serving_steps.rs:106), neither calls archive_service_log nor stop_scenario_services.
Two consequences. The engine's startup banner dies with the scenario's TempDir, so the most useful part of the diagnostic is gone by the time anyone reads the failure. And more importantly, a failing run leaves a 0.0.0.0-bound vLLM still holding the GPU — that's the part I'd actually fix, since a leaked GPU-holding process can poison whatever runs next on the lane.
| "expected the CLI to say how to send the key, got:\n{output}" | ||
| ); | ||
| assert!( | ||
| key.len() >= 16, |
There was a problem hiding this comment.
Nit: this assertion can't fail. ENDPOINT_API_KEY_LEN is a constant 48, so key.len() >= 16 is true by construction, and the properties that could vary — randomness and alphabet — are already covered by generate_endpoint_api_key_is_random_and_alphanumeric.
Beyond being dead weight, it reads as a key-strength guarantee that the Gherkin never promises, which is a little misleading about what this scenario is pinning. I'd just drop it.
…nd-selectors Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/features/networking.feature
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head 2e739a5b
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. Implements the live-endpoint half of the EAI-7409 public-bind authentication contract (the pre-flight refusal was pinned in an earlier PR); not the rest of EAI-8072.
Blocking
networking-05's newserve_public_bindstep is rejected by the CLI's own pre-flight validation and can never reach its assertions —tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs(theserve_public_bind#[when], invokingrun_rocm_ok(world, &["serve", model, "--engine", engine, "--managed", "--host", "0.0.0.0", "--allow-public-bind"])),apps/rocm/src/serve_cmd.rs:527-530,apps/rocm/src/serve_port.rs:110-120,crates/rocm-core/src/process.rs:713-715.
The invocation omits--port.serve_cmd.rsunconditionally callsvalidate_port_request(&host, PortRequest::from_flag(port))before any engine work;validate_port_requestbails with"needs an explicit --port"wheneverPortRequest::is_auto()and the host is notis_canonical_loopback_host, which istrueonly for the literal string127.0.0.1(process.rs:713-715, doc comment: "localhost,::1,0.0.0.0, and other IPv4 addresses are 'custom hosts': they require an explicit port").0.0.0.0is therefore non-canonical, so this step hits the exact refusal path the sibling scenarionetworking-04("A non-canonical host without an explicit port is refused up front") deliberately exercises for a different host.run_rocm_okpanics on non-zero exit, so the scenario fails at theWhenstep, before any key is issued and before any of the fourThenassertions run. I confirmedserve_port.rs,process.rsand the relevantserve_cmd.rslines are byte-identical between this PR's head and currentmain, and that the step's args have never included--portin any commit since the scenario was introduced (a6559658), so this is not an artifact of an older/different revision.
This also means the PR's own description ("MI300X dispatch earlier on this scenario: green (1 scenario, 6/6 steps)") anddocs/testing.md's new paragraph (claiming the scenario "checks that a request without the key and one with a wrong key get 401, while the issued key is accepted") describe a result the current code cannot produce — the self-hosted "E2E tests (MI300X, current)" check, which is the only one that would actually run this scenario, is still queued/pending against this exact head at review time, so there is no green run of this commit to reconcile this against. The "deferred follow-up" languagedocs/testing.mdreplaced would still be accurate today.
Confidence 92 · logic · Fix: pass an explicit--portinserve_public_bind(the sharedSERVE_PORT/SERVE_BASE_URLalready used by every other GPU serve scenario inserving_steps.rsis the natural choice, sincetry_models_status/get_models_statusalready assume that port), then re-verify this exact commit on the MI300X lane before merging.
Non-blocking
None
Decisions for the author
None
Positive signals
wait_for_authenticated_endpointcarefully distinguishes "connection refused while the engine is still binding" (retry) from a definitive401(fail fast, since a bound engine rejecting its own issued key is a contract defect, not a slow start) from any other status — a real, previously-hit failure mode is called out in the comments and handled correctly.- The "shown exactly once / no leak" assertions in
shows_key_oncecorrectly special-case the intentionalexample: curl ...line so it isn't double-counted as a second disclosure or a leak. docs/architecture.md's new table row is accurate and correctly alphabetized among its neighbors.- The
@requires-engine:vllmtag rationale in the feature-file comment (Windows/lemonade deliberately refuses a public bind) checks out againstapps/rocm/src/service_launch/endpoint_auth.rs'sensure_public_bind_engine_supported.
Deployment notes
This scenario only runs on the self-hosted MI300X lane and is not a required check, so a regression here (including the one above) can sit unnoticed for a while if that lane is backed up — as it is on this very PR, per its own test-plan note about a prior runner-side failure and the currently-queued re-run.
What this covered
Read every changed file in full (docs/architecture.md, docs/testing.md, tests/e2e-cucumber/features/networking.feature, tests/e2e-cucumber/tests/e2e.rs, tests/e2e-cucumber/tests/e2e/serving_steps.rs, and the new tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs), plus the production code the new test drives and asserts against (apps/rocm/src/serve_cmd.rs, apps/rocm/src/serve_port.rs, crates/rocm-core/src/process.rs, apps/rocm/src/service_launch/endpoint_auth.rs, apps/rocm/src/service_launch/start.rs, apps/rocm/src/serve_summary.rs), the full networking.feature file for context, and the CI status roll-up for this head. Diffed be9ede5d1…2e739a5b.
A second independent pass over the same scope was run in parallel and reported the same root-cause defect by an independent trace; its exact wording was not used as evidence, only as corroboration, consistent with reaching this finding blind. Did not run: "prior changes" pass (no prior review threads were read, per method). Agent-instruction pass found no AGENTS.md rule this change violates. History pass: confirmed via git log -p that the missing --port has been present since the scenario's first commit. Not reconciled against any PR discussion thread.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change.
Pins the second half of the public-bind endpoint authentication contract (EAI-7409) — key issuance plus live 401/401/2xx enforcement from a real endpoint — where an earlier PR already covered the pre-flight opt-in refusal; the ticket itself lives on a tracker not reachable from here, so this is taken only from the PR description.
Blocking
- The new GPU scenario fails on the only CI lane that can run it, on this PR's exact current head, because the new step invocation never passes
--port—tests/e2e-cucumber/tests/e2e/endpoint_auth_steps.rs:118-130(inline comment added)serve_public_bind()runsrocm serve <model> --engine <engine> --managed --host 0.0.0.0 --allow-public-bindwith no--port.apps/rocm/src/serve_port.rs's own policy — and the siblingnetworking-04scenario added earlier on this same branch — establish that automatic port selection is accepted only on the canonical loopback host127.0.0.1; any other host, including0.0.0.0, requires an explicit--port.- I read the actual
E2E tests (MI300X, current)job log for head2e739a5b(run 43422777319, job 131876765826):networking-05panics immediately withError: \rocm serve --host 0.0.0.0` needs an explicit `--port <1-65535>`` — before the CLI ever issues a key. None of the behavior this PR exists to pin (key issuance, 401 without a key, 401 with the wrong key, 2xx with the issued key) is currently exercised on real hardware. - That job's GitHub conclusion reads
CANCELLED, notFAILED, only because it was killed roughly 90 minutes later during an unrelated, later scenario in the same long-running suite — so this does not surface as a red required check, but the scenario itself unambiguously failed on this head. - Confidence 100 (direct evidence: read the job's own log for this exact head commit) · logic · Fix: pass an explicit
--port(e.g. the sharedSERVE_PORT/11435thatensure_serve_port_freeand the readiness helpers already assume) in the args list.
Non-blocking
None
Decisions for the author
None
Positive signals
wait_for_authenticated_endpointcleanly separates "connection refused while the engine is still binding" (tolerated, retried) from "the issued key itself got 401 from a bound endpoint" (fatal, reported immediately) — avoids treating every transport error the same way during a slow GPU serve.shows_key_once's leak check counts the dedicatedapi key:line rather than raw occurrences of the key string, correctly tolerating the key's deliberate second appearance inside the printed curl example rather than false-flagging it as a leak.- The feature-file comments explain why
@requires-engine:vllmis load-bearing (Windows lemonade can't enforce the key, so an untagged scenario would fail there) rather than just applying the tag silently.
Deployment notes
None — test-only change. The three helpers widened to pub(crate) in serving_steps.rs carry no behavior change.
What this covered
Read the full diff (be9ede5d1…2e739a5b, all 6 changed files) completely, including the new endpoint_auth_steps.rs in full, plus the surrounding code it depends on: serving_steps.rs (SERVE_BASE_URL, host_serve_target, serve_timeout_for, ensure_serve_port_free, run_rocm_ok), serve_port.rs's port-selection policy, service_launch/start.rs's key-printing format (confirmed the test's string assertions match the real output), feature_naming.rs's per-feature sequential-id check, and the full networking.feature. Checked the PR's 10-commit history, which already fixed several earlier review rounds' findings (scenario naming that broke feature_naming, the key-leak check's false positive on the curl example). Read CI directly via the GitHub API and job logs rather than trusting the green rollup: every required check is green, but the one self-hosted GPU job that actually exercises networking-05 ran against this exact head and failed the new step, a failure the rollup doesn't surface because that lane isn't a required status check and the job's later cancellation (unrelated scenario, ~90 min on) reports as CANCELLED rather than FAILED. Did not run anything locally — no GPU/engine available here — so the CI log is the verification. Ticket EAI-7409 is on a tracker not reachable from here; its criteria are taken only from the PR description.
Single consolidated pass rather than a multi-worker fan-out, given the diff's modest size (276 additions across 6 files, one new ~220-line file) was within what one reviewer could read completely end to end. The "prior changes" pass (separate historical PRs' review comments on these files) surfaced nothing beyond what this PR's own commit history already shows being iterated on. Not reconciled against this PR's existing review-comment discussion — none of it was read for this pass.
🤖 Automated by an AI agent
| let stdout = crate::run_rocm_ok( | ||
| world, | ||
| &[ | ||
| "serve", | ||
| model, | ||
| "--engine", | ||
| engine, | ||
| "--managed", | ||
| "--host", | ||
| "0.0.0.0", | ||
| "--allow-public-bind", | ||
| ], | ||
| ); |
There was a problem hiding this comment.
Blocking — this invocation fails before the CLI ever issues a key, and the GPU CI log for this exact head confirms it.
No --port is passed, but --host 0.0.0.0 is not the canonical loopback host, so serve_port.rs's own policy (and the sibling networking-04 scenario added on this branch) requires an explicit --port on any non-canonical host — automatic selection is refused there.
I read the E2E tests (MI300X, current) job for this PR's current head 2e739a5b (run 43422777319, job 131876765826). networking-05 fails immediately:
Step panicked. Captured output: `rocm serve Qwen/Qwen3.5-0.8B --engine vllm --managed --host 0.0.0.0 --allow-public-bind` failed (rc=1)
--- stderr ---
Error: `rocm serve --host 0.0.0.0` needs an explicit `--port <1-65535>`. Automatic port selection is supported only on the default host 127.0.0.1, ...
So on real hardware right now, none of the behavior this PR exists to pin — key issuance, 401 without a key, 401 with the wrong key, 2xx with the issued key — is actually exercised. The job's GitHub conclusion shows CANCELLED rather than FAILED only because it was killed ~90 minutes later during an unrelated later scenario, so this failure doesn't surface as a red required check, but the scenario itself unambiguously failed.
Fix: pass an explicit --port in this args list (e.g. the shared SERVE_PORT / 11435 that ensure_serve_port_free/the readiness helpers already assume).
Summary
Pins the public-bind endpoint authentication contract (EAI-7409) end to end: with
--allow-public-bind, the CLI issues a key once with Bearer guidance, and the live endpoint answers 401 without a key, 401 with a wrong key, and 2xx with the issued key. This was the highest-value item left from EAI-8072; the pre-flight refusal was already pinned in the earlier PR.The scenario is
@requires-gpu @requires-engine:vllm, so it skips on Windows/lemonade hosts (where the CLI deliberately refuses a public bind) and runs only on MI300X. Readiness probing retries while the engine binds, and a timeout reports serve output, service log tail and the last endpoint response.Test plan